Methods and systems for quality-aware continuous learning for radiotherapy treatment planning

ABSTRACT

Example methods and systems for quality-aware continuous learning for radiotherapy treatment planning are provided. One example method may comprise: obtaining an artificial intelligence (AI) engine that is trained to perform a radiotherapy treatment planning task. The method may also comprise: based on input data associated with a patient, performing the radiotherapy treatment planning task using the AI engine to generate output data associated with the patient; and obtaining modified output data that includes one or more modifications made by a treatment planner to the output data. The method may further comprise: performing quality evaluation based on (a) first quality indicator data associated with the modified output data, and/or (b) second quality indicator data associated with the treatment planner. In response to a decision to accept, a modified AI engine may be generated by re-training the AI engine based on the modified output data.

BACKGROUND

Unless otherwise indicated herein, the approaches described in thissection are not prior art to the claims in this application and are notadmitted to be prior art by inclusion in this section.

Radiotherapy is an important part of a treatment for reducing oreliminating unwanted tumors from patients. Unfortunately, appliedradiation does not inherently discriminate between an unwanted tumor andany proximal healthy structures such as organs, etc. This necessitatescareful administration to restrict the radiation to the tumor (i.e.,target). Ideally, the goal is to deliver a lethal or curative radiationdose to the tumor, while maintaining an acceptable dose level in theproximal healthy structures. However, to achieve this goal, conventionalradiotherapy treatment planning may be time and labor intensive.

SUMMARY

According to examples of the present disclosure, methods and systems forquality-aware continuous learning for radiotherapy treatment planningare provided. In this case, one example method may comprise: obtainingan artificial intelligence (AI) engine that is trained to perform aradiotherapy treatment planning task. The method may also comprise:based on input data associated with a patient, performing theradiotherapy treatment planning task using the AI engine to generateoutput data associated with the patient; and obtaining modified outputdata that includes one or more modifications made by a treatment plannerto the output data.

The example method may further comprise: performing quality evaluationbased on (a) first quality indicator data associated with the modifiedoutput data, and/or (b) second quality indicator data associated withthe treatment planner. In response to a decision to accept, a modifiedAI engine may be generated by re-training the AI engine based on themodified output data.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic diagram illustrating an example process flow forradiotherapy treatment;

FIG. 2 is a flowchart of an example process for a computer system toperform quality-aware continuous learning for radiotherapy treatmentplanning;

FIG. 3 is a schematic diagram illustrating example quality-awarecontinuous learning for radiotherapy treatment planning according to theexample in FIG. 2;

FIG. 4 is a schematic diagram illustrating example quality-awarecontinuous learning for automatic segmentation;

FIG. 5 is a flowchart of an example process for quality evaluation tofacilitate quality-aware continuous learning for radiotherapy treatmentplanning;

FIG. 6 is a schematic diagram illustrating an example process forcredibility score assignment to facilitate quality-aware continuouslearning for radiotherapy treatment planning;

FIG. 7 is a schematic diagram illustrating example quality-awarecontinuous learning for dose prediction;

FIG. 8 is a schematic diagram illustrating example quality-awarecontinuous learning for radiotherapy treatment planning using amulti-technique AI engine;

FIG. 9 is a flowchart of an example process for expert panel selection,planning task assignment and reward determination based on credibilityscore;

FIG. 10 is a schematic diagram of an example treatment plan fortreatment delivery; and

FIG. 11 is a schematic diagram of an example computer system to performquality-aware continuous learning for radiotherapy treatment planning.

DETAILED DESCRIPTION

The technical details set forth in the following description enable aperson skilled in the art to implement one or more embodiments of thepresent disclosure.

FIG. 1 is a schematic diagram illustrating example process flow 100 forradiotherapy treatment. Example process 100 may include one or moreoperations, functions, or actions illustrated by one or more blocks. Thevarious blocks may be combined into fewer blocks, divided intoadditional blocks, and/or eliminated based upon the desiredimplementation. In the example in FIG. 1, radiotherapy treatmentgenerally includes various stages, such as an imaging system performingimage data acquisition for a patient (see 110); a radiotherapy treatmentplanning system (see 130) generating a suitable treatment plan (see 156)for the patient; and a treatment delivery system (see 160) deliveringtreatment according to the treatment plan.

In more detail, at 110 in FIG. 1, image data acquisition may beperformed using an imaging system to capture image data 120 associatedwith a patient (particularly the patient's anatomy). Any suitablemedical image modality or modalities may be used, such as computedtomography (CT), cone beam computed tomography (CBCT), positron emissiontomography (PET), magnetic resonance imaging (MRI), single photonemission computed tomography (SPECT), any combination thereof, etc. Forexample, when CT or MRI is used, image data 120 may include a series oftwo-dimensional (2D) images or slices, each representing across-sectional view of the patient's anatomy, or may include volumetricor three-dimensional (3D) images of the patient, or may include a timeseries of 2D or 3D images of the patient (e.g., four-dimensional (4D) CTor 4D CBCT).

At 130 in FIG. 1, radiotherapy treatment planning may be performedduring a planning phase to generate treatment plan 156 based on imagedata 120. Any suitable number of treatment planning tasks or steps maybe performed, such as segmentation, dose prediction, projection dataprediction, treatment plan generation, etc. For example, segmentationmay be performed to generate structure data 140 identifying varioussegments or structures from image data 120. In practice, athree-dimensional (3D) volume of the patient's anatomy may bereconstructed from image data 120. The 3D volume that will be subjectedto radiation is known as a treatment or irradiated volume that may bedivided into multiple smaller volume-pixels (voxels) 142. Each voxel 142represents a 3D element associated with location (i, j, k) within thetreatment volume. Structure data 140 may be include any suitable datarelating to the contour, shape, size and location of patient's anatomy144, target 146, organ-at-risk (OAR) 148, etc. In practice, OAR 148 mayrepresent any suitable delineated organ, non-target structure (e.g.,bone, tissue, etc.), etc.

For example, using image segmentation, a line may be drawn around asection of an image and labelled as target 146 (e.g., tagged withlabel=“prostate”). Everything inside the line would be deemed as target146, while everything outside would not. In another example, doseprediction may be performed to generate dose data 150 specifyingradiation dose to be delivered to target 146 (denoted “D_(TAR)” at 152)and radiation dose for OAR 148 (denoted “D_(OAR)” at 154). In practice,target 146 may represent a malignant tumor (e.g., prostate tumor, etc.)requiring radiotherapy treatment, and OAR 148 a proximal healthystructure or non-target structure (e.g., rectum, bladder, etc.) thatmight be adversely affected by the treatment. Target 146 is also knownas a planning target volume (PTV). Although an example is shown in FIG.1, the treatment volume may include multiple targets 146 and OARs 148with complex shapes and sizes. Further, although shown as having aregular shape (e.g., cube), voxel 142 may have any suitable shape (e.g.,non-regular). Depending on the desired implementation, radiotherapytreatment planning at block 130 may be performed based on any additionaland/or alternative data, such as prescription, disease staging, biologicor radiomic data, genetic data, assay data, biopsy data, past treatmentor medical history, any combination thereof, etc.

Based on structure data 140 and dose data 150, treatment plan 156 may begenerated to include 2D fluence map data for a set of beam orientationsor angles. Each fluence map specifies the intensity and shape (e.g., asdetermined by a multileaf collimator (MLC)) of a radiation beam emittedfrom a radiation source at a particular beam orientation and at aparticular time. For example, in practice, intensity modulatedradiotherapy treatment (IMRT) or any other treatment technique(s) mayinvolve varying the shape and intensity of the radiation beam while at aconstant gantry and couch angle. Alternatively or additionally,treatment plan 156 may include machine control point data (e.g., jaw andleaf positions), volumetric modulated arc therapy (VMAT) trajectory datafor controlling a treatment delivery system, etc. In practice, block 130may be performed based on goal doses prescribed by a clinician (e.g.,oncologist, dosimetrist, planner, etc.), such as based on theclinician's experience, the type and extent of the tumor, patientgeometry and condition, etc.

At 160 in FIG. 1, treatment delivery is performed during a treatmentphase to deliver radiation to the patient according to treatment plan156. For example, radiotherapy treatment delivery system 160 may includerotatable gantry 164 to which radiation source 166 is attached. Duringtreatment delivery, gantry 164 is rotated around patient 170 supportedon structure 172 (e.g., table) to emit radiation beam 168 at variousbeam orientations according to treatment plan 156. Controller 162 may beused to retrieve treatment plan 156 and control gantry 164, radiationsource 166 and radiation beam 168 to deliver radiotherapy treatmentaccording to treatment plan 156. The radiation may be designed to becurative, palliative, adjuvant, etc.

It should be understood that any suitable radiotherapy treatmentdelivery system(s) may be used, such as mechanic-arm-based systems,tomotherapy type systems, brachytherapy, SIR-spheres,radiopharmaceuticals, any combination thereof, etc. Additionally,examples of the present disclosure may be applicable to particledelivery systems (e.g., proton, carbon ion, etc.). Such systems mayemploy either a scattered particle beam that is then shaped by a deviceakin to an MLC, or a scanning beam of adjustable energy, spot size anddwell time.

Conventionally, radiotherapy treatment planning at block 130 in FIG. 1is time and labor intensive. For example, it usually requires a team ofhighly skilled and trained oncologists and dosimetrists to manuallydelineate structures of interest by drawing contours or segmentations onimage data 120. These structures are manually reviewed by a physician,possibly requiring adjustment or re-drawing. In many cases, thesegmentation of critical organs can be the most time-consuming part ofradiation treatment planning. After the structures are agreed upon,there are additional labor-intensive steps to process the structures togenerate a clinically-optimal treatment plan specifying treatmentdelivery data such as beam orientations and trajectories, as well ascorresponding 2D fluence maps.

Further, treatment planning is often complicated by a lack of consensusamong different physicians and/or clinical regions as to whatconstitutes “good” contours or segmentation. In practice, there might bea huge variation in the way structures or segments are drawn bydifferent clinical experts. The variation may result in uncertainty intarget volume size and shape, as well as the exact proximity, size andshape of OARs that should receive minimal radiation dose. Even for aparticular expert, there might be variation in the way segments aredrawn on different days. Due to the lack of consistency, treatmentplanning might result in different clinical outcomes for patients and itis difficult to evaluate whether a final treatment plan is “good.”

According to examples of the present disclosure, artificial intelligence(AI) techniques may be applied to ameliorate various challengesassociated with radiotherapy treatment planning. Throughout the presentdisclosure, the term “AI engine” may refer generally to any suitablehardware and/or software component(s) of a computer system capable ofexecuting algorithms according to any suitable AI model(s), such as deeplearning model(s), etc. The term “deep learning” may refer generally toa class of approaches that utilizes many layers or stages of nonlineardata processing for feature learning as well as pattern analysis and/orclassification. The “deep learning model” may refer to a hierarchy of“layers” of nonlinear data processing that include an input layer, anoutput layer, and multiple (i.e., two or more) “hidden” layers betweenthe input and output layers. These layers may be trained from end-to-end(e.g., from the input layer to the output layer) to extract feature(s)from an input and classify the feature(s) to produce an output (e.g.,classification label or class). The term “deep learning engine” mayrefer to any suitable hardware and/or software component(s) of acomputer system capable of executing algorithms according to anysuitable deep learning model(s).

Depending on the desired implementation, any suitable AI model(s) may beused, such as convolutional neural network, recurrent neural network,deep belief network, or any combination thereof, etc. In practice, aneural network is generally formed using a network of processingelements (called “neurons,” “nodes,” etc.) that are interconnected viaconnections (called “synapses,” “weights,” etc.). For example,convolutional neural networks may be implemented using any suitablearchitecture(s), such as U-net, LeNet, AlexNet, ResNet, V-net, DenseNet,etc. In this case, a “layer” of a convolutional neural network may be aconvolutional layer, pooling layer, rectified linear units (ReLU) layer,fully connected layer, loss layer, etc. In practice, the U-netarchitecture includes a contracting path (left side) and an expansivepath (right side). The contracting path includes repeated application ofconvolutions, followed by a ReLU layer and max pooling layer. Each stepin the expansive path may include upsampling of the feature map followedby convolutions, etc.

Deep learning approaches should be contrasted against machine learningapproaches that have been applied to, for example, automaticsegmentation. In general, these approaches involve extracting(hand-designed) feature vectors from images, such as for every voxel,etc. Then, the feature vectors may be used as input to a machinelearning model that classifies which class each voxel belongs to.However, such machine learning approaches usually rely on a highdimension of hand-designed features in order to accurately predict theclass label for each voxel. Solving a high-dimensional classificationproblem is computationally expensive and requires a large amount ofmemory. Some approaches use lower dimensional features (e.g., usingdimensionality reduction techniques) but they may decrease theprediction accuracy.

Conventionally, there are many challenges associated with training AIengines (e.g., deep learning engines) for radiotherapy treatmentplanning. For example, different planners generally have differentclinical practices in radiotherapy treatment planning. To train an AIengine according to a specific clinical practice, one option is todevelop a specific in-house model. However, it may be difficult toachieve desirable training results without collecting a huge amount ofcarefully-curated training data. Also, while conceptually simple,training AI engines generally requires significant technical expertiserelating to model architecture(s), optimization, convergence analysis,regularization, etc. These challenges may lead to suboptimal results or,worse, failure to create any working AI engines. Such complexity maydeter users from training and using AI engines for radiotherapytreatment planning, which is undesirable.

Quality-Aware Continuous Learning

According to examples of the present disclosure, quality-awarecontinuous learning may be implemented to improve the performance of AIengines for radiotherapy treatment planning. As used herein, the term“continuous learning” (also known as “lifelong learning,” “incrementallearning” and “sequential learning”) may refer generally to technique(s)where an AI engine is modified or improved throughout its operationbased on additional training data. The term “quality-aware” may refergenerally to a quality evaluation process for deciding whethercontinuous learning should be performed.

Using a quality-aware approach, a trained AI engine may be modified orimproved over time based on training data that has been evaluated. Byimproving the quality and adaptability of AI engines, treatment planningoutcome may also be improved for patients, such as increasing the tumorcontrol probability and/or reducing the likelihood of healthcomplications or death due to radiation overdose in the healthystructures, etc. Examples of the present disclosure may be deployed inany suitable manner, such as a standalone computer system, web-basedplanning-as-a-service (PaaS) system, or any combination thereof, etc.

In more detail, FIG. 2 is a flowchart illustrating example process 200for a computer system to perform quality-aware continuous learning forradiotherapy treatment planning. Example process 200 may include one ormore operations, functions, or actions illustrated by one or moreblocks, such as 210 to 270. The various blocks may be combined intofewer blocks, divided into additional blocks, and/or eliminated basedupon the desired implementation. Example process 200 may be implementedusing any suitable computer system(s), an example of which will bediscussed using FIG. 11. Some examples will be explained using FIG. 3,which is a schematic diagram illustrating example quality-awarecontinuous learning for radiotherapy treatment planning according to theexample in FIG. 2.

At 210 in FIG. 2, a treatment planning AI engine (see 320 in FIG. 3)that is trained to perform a radiotherapy treatment planning task may beobtained. Here, the term “obtain” may refer generally to a computersystem accessing, or retrieving data and/or computer-readableinstructions associated with, AI engine 320 from any suitable source(e.g., another computer system), memory or datastore (e.g., local orremote), etc. AI engine 320 may be trained during training phase 301based on first training data (see 310 in FIG. 3) that includes treatmentplans associated with multiple past patients. Note that “first trainingdata” 310 may include synthetic training data, which are training casesderived from past patients.

At 220 in FIG. 2, AI engine 320 may be used to perform the radiotherapytreatment planning task during inference phase 302. For example, basedon input data (see 330 in FIG. 3) associated with a particular patient,AI engine 320 may perform the radiotherapy treatment planning task togenerate output data (see 340 in FIG. 3) associated with the patient. Inpractice, AI engine 320 may be trained to perform any suitableradiotherapy treatment planning task, such as automatic segmentation,dose prediction, treatment delivery data estimation, abnormal organdetection, treatment outcome prediction, or any combination thereof.

In the case of automatic segmentation, AI engine 320 may be trained togenerate output=structure data (e.g., 140 in FIG. 1) based oninput=image data (e.g., 120 in FIG. 1). In the case of dose prediction,engine 320 may be trained to generate output=dose data (e.g., 150 inFIG. 1) based on input=structure data and beam geometry data. In thecase of treatment delivery data estimation, engine 320 may be trained togenerate output=treatment delivery data (e.g., fluence map data,structure projection data, etc.) based on input=structure data and/ordose data, etc.

At 230 in FIG. 2, modified output data (see 350 in FIG. 3) that includesmodification(s) to output data 340 may be obtained. The term“modification” may refer generally to an addition, deletion, correction,change, movement, selection or alteration that may be made to outputdata. Modified output data 350 may be generated by a treatment planner(see 355 in FIG. 3). Here, a “treatment planner” or “planner” may refergenerally to an individual, a group of individuals, an institution, aclinical site or network, a clinical region, or any combination thereof.For example, an individual may be a dosimetrist, clinician, physicist,medical personnel, etc. In some situations, a “treatment planner” may beanother computerized algorithm.

In practice, the modification(s) may be made by a treatment planner 355according to any suitable clinical guideline(s), planning strategyand/or planning practice(s) associated with the treatment planner. Forexample, in the case of automatic segmentation (to be discussed usingFIG. 4), modified output data 350 may include a modification made tosegmentation margins associated with a structure (e.g., OAR or target).In relation dose prediction (to be discussed using FIG. 7), modifiedoutput data 350 may include a modification made to OAR sparing, etc. Anyalternative and/or additional modification(s) may be used.

At 240 in FIG. 2, quality evaluation of modified output data 350 may beperformed based on (a) first quality indicator data (see 360/361 in FIG.3) associated with modified output data 350 and/or (b) second qualityindicator data (see 360/362 in FIG. 3) associated with treatment planner355. As used herein, the term “quality indicator data” may refergenerally to any qualitative and/or quantitative factor(s) orvariable(s) that help inform a decision process as to whether to performcontinuous learning based on modified output data 350.

In the example in FIG. 3, block 240 may include determining firstquality indicator data in the form of statistical model parameter data(see 361 in FIG. 3) by applying statistical model(s) on modified outputdata 350. Additionally and/or alternatively, block 240 may includeidentifying the i^(th) treatment planner from multiple planners, anddetermining second quality indicator data in the form of a credibilityscore C(i) assigned to the i^(th) treatment planner (see 362 in FIG. 3).The term “credibility score” (to be discussed further using FIG. 6) mayrefer generally to any suitable quantitative measure that represents areputation or trustworthiness of a particular treatment planner.

At 250 in FIG. 2, a decision may be made as to whether to acceptmodified output data 350 for continuous learning based on the qualityevaluation at block 240. At 260 in FIG. 2, in response to a decision toaccept (see 370 in FIG. 3), modified AI engine (see 390 in FIG. 3) maybe generated by re-training AI engine 320 based on modified output data360 during continuous learning phase 303. Otherwise, at 270 in FIG. 2,continuous learning based on modified output data 360 will not beperformed (see also 375 in FIG. 3). In practice, the re-training processat block 260 may involve modifying or improving weight data associatedwith AI engine 320. Further, block 260 may involve generating secondtraining data (see 380 in FIG. 3) based on modified output data 360 forthe re-training process. A case weight may also be assigned to modifiedoutput data 360 based on (a) first quality indicator data 361 and/or (b)second quality indicator data 362 to influence the re-training process.

Depending on the desired implementation, the re-training process atblock 260 may be performed for a batch of modified output data 350. Forexample, the batch may include modification(s) made by multipletreatment planners over a period of time. For quality assurancepurposes, the re-training process may be performed periodically so thatre-trained or modified AI engine 350 may undergo some form of qualityassurance checks prior to deployment.

According to examples of the present disclosure, continuous learningphase 303 may be improved using additional training data that has beenevaluated for quality. If rejected, continuous learning will not beperformed, thereby improving efficiency and reducing the negative impactof inferior or redundant training data. Various examples will bediscussed below using FIG. 4 to FIG. 11. In particular, an exampleautomatic segmentation will be discussed using FIG. 4, an examplequality evaluation using FIG. 5, an example credibility score assignmentusing FIG. 6, an example dose prediction using FIG. 7, an examplemulti-technique AI engine using FIG. 8, various use cases forcredibility scores using FIG. 9, an example treatment plan using FIG.10, and an example computer system using FIG. 11.

Automatic Segmentation Example

FIG. 4 is a schematic diagram illustrating example quality-awarecontinuous learning for automatic segmentation 400. In this example, AIengine 420 (also referred to as “segmentation engine” below) may betrained using first training data 410 during training phase 401; appliedto perform automatic segmentation during inference phase 402; andupdated during continuous learning phase 403. In practice, the output ofautomatic segmentation may be used for abnormal organ detection, doseprediction, treatment delivery data estimation, etc.

(a) Training Phase (See 401 in FIG. 4)

During training phase 401, segmentation engine 420 may be trained to maptraining image data 411 (i.e., input) to training structure data 412(i.e., output). In practice, image data 411 may include 2D or 3D imagesof a patient's anatomy, and captured using any suitable imaging modalityor modalities. Structure data 412 may identify any suitable contour,shape, size and/or location of structure(s) from image data 411. Examplestructures may include target(s), OAR(s) or any other structure ofinterest (e.g., tissue, bone) of the anatomical site. Depending on thedesired implementation, structure data 412 may identify multiple targetsand OARs of any suitable shapes and sizes.

For example, in relation to prostate cancer, image data 411 may includeimages of site=prostate region. In this case, structure data 412 mayidentify a target representing each patient's prostate, and OARsrepresenting proximal healthy structures such as rectum and bladder. Inrelation to lung cancer treatment, image data 411 may include images ofa lung region. In this case, structure data 412 may identify a targetrepresenting cancerous lung tissue, and an OAR representing proximalhealthy lung tissue, esophagus, heart, etc. In relation to brain cancer,image data 411 may include images of a brain region. Structure data 412may identify a target representing a brain tumor, and an OARrepresenting a proximal optic nerve, brain stem, etc.

First training data 410 may be extracted or derived from past treatmentplans developed for multiple past patients according to any desirableplanning rule. First training data 410 may be pre-processed using anysuitable data augmentation approach (e.g., rotation, flipping,translation, scaling, noise addition, cropping, any combination thereof,etc.) to produce a new dataset with modified properties to improve modelgeneralization using ground truth. In practice, a 3D volume of thepatient that will be subjected to radiation is known as a treatmentvolume, which may be divided into multiple smaller volume-pixels(voxels). In this case, structure data 412 may specify a class label(e.g., “target,” “OAR,” etc.) associated with each voxel in the 3Dvolume.

In one example in FIG. 4, segmentation engine 420 includes multiple(N>1) processing blocks or layers that are each associated with a set ofweight data. In this case, training phase 401 may involve finding weightdata that minimizes a training error between training structure data412, and estimated structure data (not shown for simplicity) generatedby segmentation engine 420. The training process is guided by estimatinglosses associated with the classification error. A simple example of aloss function would be mean squared error between true and predictedoutcome, but the loss function could have more complex formulas. Thisloss can be estimated from the output of the model, or from any discretepoint within the model.

(b) Inference Phase (See 402 in FIG. 4)

At 430 and 440 in FIG. 4, trained segmentation engine 420 may be used toperform automatic segmentation for a particular patient during inferencephase 402. Input image data 430 associated with that patient isprocessed using segmentation engine 420 to generate output structuredata 440. For example, output structure data 440 may identify anysuitable contour, shape, size and/or location of structure(s) in inputimage data 430.

At 450 in FIG. 4, output structure data 440 may be modified by thetreatment planner 455 to achieve a preferred segmentation outcome. Forexample, modified output structure data 450 may include modification(s)made by planner 455 to the contour, edges, shape, size and/or locationof structure(s) in output structure data 440. In the case of automaticsegmentation, a modification may refer to moving, adjusting or redrawingsegmentation(s). For example, modified output structure data 450 mayinclude different segmentation margin(s), identify an additional and/oralternative structure, etc.

(c) Continuous Learning Phase (See 403 in FIG. 4)

At 461-462 and 470 in FIG. 4, a quality evaluation may be performedbased on any suitable quality indicator data to decide whether to acceptmodified output data 450 for continuous learning. As will be discussedfurther using FIG. 5, the quality evaluation may be based on statisticalparameter data 461 (“first quality indicator data”) that is generated byidentifying and applying statistical model(s) on modified output data450. Alternatively and/or additionally, credibility score 462 (“secondquality indicator data”) associated with treatment planner 455 may beobtained.

In one example, the quality evaluation may involve performing a firstfiltering of modified output data 450 based on statistical parameterdata 461. If a first threshold is satisfied (and credibility score 462is available), a second filtering is then performed based on credibilityscore 462. In this case, statistical parameter data 461 may be used todetermine whether modification(s) made by treatment planner 455 provideany measurable improvement according to the statistical model(s). Whencombined with credibility score 462 associated with treatment planner455 that made the modification, modified output data 450 may becategorized to be “high value” (i.e., decision=ACCEPT) or “low value”(i.e., decision=REJECT). Some examples will be discussed below usingFIG. 5.

At 480 and 490 in FIG. 4, in response to a decision to accept (see 472)modified output data 450 for continuous learning based on the qualityevaluation, segmentation engine 420 may be updated based on secondtraining data 480. In practice, second training data 480 may includeexample input-output pair in the form of image data 430 processed bysegmentation engine 420, and modified output structure data 450 thatincludes modification(s) desired by planner 455. Once continuouslearning is performed, modified segmentation engine 480 may be deployedfor use in the next iteration of inference phase 402. If modification ismade to subsequent output structure data generated by modified engine480, quality evaluation and continuous learning phase 403 may berepeated for further improvement.

Quality Evaluation

FIG. 5 is a flowchart illustrating example process 500 for qualityevaluation to facilitate quality-aware continuous learning forradiotherapy treatment planning. Example process 500 may include one ormore operations, functions, or actions illustrated by one or moreblocks, such as 510 to 570. The various blocks may be combined intofewer blocks, divided into additional blocks, and/or eliminated basedupon the desired implementation. Example process 500 may be implementedusing any suitable computer system(s), an example of which will bediscussed using FIG. 11.

In practice, any suitable quality indicator data denoted as Qk (wherek=1, . . . , K) may be used for quality evaluation. As will be describedfurther below, the quality indicator data may include statisticalparameter data (see 510), credibility score (see 520), expert reviewdata (see 530), etc. This way, at 540, quality evaluation of modifiedoutput data 450 may be performed based on any combination of qualityindicator data (Q1, . . . , QK).

(a) Statistical Parameter Data

At 510 in FIG. 5, first quality indicator data (Q1 for k=1) in the formof statistical parameter data 461 may be determined. Block 510 mayinvolve identifying and applying statistical model(s) to the modifiedoutput data 450. See 512-514 in FIG. 5. In practice, the term“statistical model” may refer generally to a model for evaluating aprobability of certain measurable quantity (also known as an attribute,feature or parameter) derivable from modified output data 450. This way,statistical parameter data 461 may be used to indicate the reliabilityor validity of modified output data 450 associated with planner 455 inFIG. 4.

Any suitable “statistical model” may be used, ranging from simplemetric(s) to more complex model(s). The statistical model may return asingle value (i.e., scalar) or multiple values (vector). Further, the“probability” evaluated using a statistical model may be unconditional(e.g., describes the statistics of the quantity over the whole set oftraining data) or conditional (e.g., describes the statistics of somequantity in condition that certain previous intermediate result has beenobtained). For example, structure size could be used unconditionally(e.g., it has the same criteria regardless of the CT scan), or it couldbe conditional to some values derived from the CT scan (e.g., total areaof non-zero region in the slice going through the center of the givenorgan).

In the case of automatic segmentation in FIG. 4, statistical models maybe applied to evaluate a probability of certain attribute associatedwith a patient's structure data 450, such as its shape, size, texture,contour, principal component analysis, material density, geometricdistribution, or Hounsfield Units (HU) distribution, relative positionof the structure to another structure, etc. For example, in relation toprostate cancer treatment, statistical parameter data 461 may begenerated based on a statistical model for evaluating the sphericity ofa prostate (i.e., target).

In the case of dose prediction, statistical models may be applied toevaluate certain attribute(s) associated with dose data, such as DVH,D20, D50, dose fall-off gradient, etc. Other statistical models may beused for evaluating treatment delivery data, such as smoothness of 2Dfluence maps; total motion of leaves in VMAT plan, beam orientations,machine trajectories, any combination thereof, etc. Depending on theradiotherapy treatment planning task that an AI engine is trained toperform, any alternative and/or additional statistical model(s) that areknown in the art may be used.

(b) Credibility Score

At 520 in FIG. 5, second quality indicator data (Q2 for k=2) in the formof credibility score 462 may be determined. Block 520 may involveidentifying a particular i^(th) planner 455 responsible for themodification(s) in modified output data 450, and obtaining a credibilityscore C(i) associated with the planner. See also 522-524 in FIG. 5. Inpractice, the i^(th) planner may be identified from multiple (P>1)planners who are each assigned with credibility score C(i), where i=1, .. . , P. Any suitable approach may be used for credibility scoreassignment. Some examples will be explained using FIG. 6, which is aschematic diagram illustrating example process 600 for credibility scoreassignment to facilitate quality-aware continuous learning forradiotherapy treatment planning.

At 610 and 620, multiple (N) treatment planning cases may be selectedfor a “planning contest” that involves multiple (P) treatment planners.At 630, each i^(th) planner-generated output data D(i, j) for eachj^(th) case may be obtained, where i=1, . . . , P and j=1, . . . , N.For example, D(1, 3) refers to a first planner's (i=1) output data for athird case (j=3), and D(4, N) to a fourth planner's (i=4) output datafor the N^(th) case (j=N). As previously noted, a “planner” mayrepresent an individual, a group of individual, an institution, aclinical site or network, etc.

In practice, treatment planning cases 620 may be new cases specificallyselected for the planning contest, or historical cases (e.g., fromhistorical planning contests). These cases may be “sprinkled” into eachplanner's daily workflow. In both cases, examples of the presentdisclosure may provide a built-in procedure for cross-calibration andconsensus truth analysis for multiple patients. A given case (j) may beused in such a calibration process until adding the output data of newplanners fails to substantially affect its corresponding consensus truthG(j). The calibration process may be presented as a training program forplanners (e.g., human delineators) to help develop and maintain theirskills over time.

In a first example (see 640 in FIG. 6), credibility score C(i) may beassigned based on a comparison between planner-generated output dataD(i, j) and consensus truth data associated with cases j=1, . . . , N.For the j^(th) case, its consensus truth data may be denoted as G(j) anddetermined based on corresponding all planner-generated output data D(i,j) for that case, where i=1, . . . , P. In this case, G(j) may be anaverage or mean of output data D(i, j) for planners i=1, . . . , P.Using a normal distribution for D(i, j) as an example, the consensustruth data for the j^(th) case may be determined using G(j)=1/P Σ_(i=1)^(N) D(i, j). Any other distribution may be used. A smaller deviationwill lead to a higher credibility score, and a higher deviation to alower credibility score.

In practice, the term “ground truth” or “absolute truth” may refergenerally to the “ideal” or “optimal” output data for the j^(th) case.Since the “ground truth” may not exist, the “consensus truth” G(j) forthe j^(th) case may be determined from output data D(i, j) produced bymultiple planners to represent the “gold standard.” In this case, acredibility score may be assigned to a planner on the basis of theirability to consistently produce output data in accordance with thecurrent consensus truth. As will be discussed further below, whenmultiple clusters of practice patterns are identified, the term“consensus truth” may refer to the mean or average of a particularcluster (cohort). For example, if there are M clusters, G(j, m) mayrepresent the consensus truth of the m^(th) cluster for the j^(th) case,where m=1, . . . , M.

In a second example (see 650 in FIG. 6), credibility score C(i) may beassigned based on cluster analysis data 650 associated with D(i, j).Here, cluster analysis data 650 may identify multiple (M) clustersindicating different self-similar methodologies used by the planners. Inpractice, cluster analysis may be performed when D(i, j) for aparticular j^(th) case converges into separate clusters. In this case,the set of planners may be increased with additional planners (e.g.,P+1, P+2, and so on). Output data D(i>P, j) generated by the additionalplanners may then be added until the clusters separates from each other“cleanly” based on any suitable threshold. In this case, the credibilityscore C(i) of the i^(th) planner may be determined based on thedeviation between the planner-generated output data D(i, j) and theconsensus truth G(j, m) of a particular m^(th) cluster. As practicepatterns move over time (and the number of clusters changes), thecontinuous learning process may be led by the most credible planners whohave higher case weights, or by masses of new practitioners who emergeas a new cluster.

In a third example (see 660 in FIG. 6), credibility score C(i) may beassigned based on expert review data 660 associated with a review ofoutput data generated by each planner, such as by a panel of humanexperts. In a first approach, the panel may review D(i, j) for theplanning contest in FIG. 6 and/or each planner's historical plans toevaluate their plan quality, segmentation accuracy, etc. In a secondapproach, each time a particular planner makes a substantial correction(e.g., adjusting a contour by more than 1-2 sigma in inter-observervariability from the result) using a treatment planning system, thecorrection may be automatically flagged in the system. The correction isthen reviewed by a panel of individuals who are regarded as experts intheir field. In this case, expert review data 660 may indicate the“value” of the correction or output data associated with each planner.Based on expert review data 660, the credibility score C(i) for theplanner may be increased, or decreased.

In a fourth example (see 670 in FIG. 6), credibility score C(i) may beassigned based on algorithm comparison data 670 that evaluates adeviation between (i) planner-generated output data that includes D(i,j) and/or historical plans and (ii) algorithm-generated output data. Forexample, the selected algorithm may be designed to produce result thatis close to the consensus truth. In this case, algorithm comparison data670 may include parameter(s) that compare the planner- andalgorithm-generated output data, such as similarity measure, meandeviation, self-consistency measure, etc. If the deviation is high, alower credibility score will be assigned. Otherwise, a highercredibility score will be assigned.

To evaluate internal consistency for a particular planner, one approachis to calculate a mean deviation between the (i) planner-generatedoutput data and (ii) the algorithm-generated output data over manycases, and the offset each planner's result by the mean deviation. Theaverage deviation from the offset (over many cases) may be used as ameasure of self-consistency. High self-consistency and a notable meandeviation (offset) from the algorithm indicate a behavioral bias orcluster. Another approach to evaluate internal consistency is to performa clustering analysis of a planner's historical plans. For example,historical plans may be divided into multiple tranches based on anysuitable factor(s), such as time (AM or PM), month, year, geographicallocation, etc. Multiple AI engines may be trained using training dataassociated with respective trances. Once trained, the AI engines may beused to process a set of test cases to evaluate whether they converge ordiverge, such as based on a temporal factor, etc. A convergence wouldindicate high internal consistency, whereas a divergence would indicatelow internal consistency.

At 680 in FIG. 6, a credibility score C(i) may be assigned to eachi^(th) planner based on any combination of the following: consensustruth data 640, cluster analysis data 650, expert review data 660,algorithm comparison data 670, etc. The examples in FIG. 6 may berepeated at any suitable frequency to update the credibility scoreassigned to each planner. Over time, good performers will have a highercredibility score compared to poor performers. Planners may be comparedbased on their respective credibility scores, such as how they stand inpercentiles compared to others.

(c) Accept or Reject Decision

Referring to FIG. 5 again, at 530, any additional and/or alternativequality indicator data may be determined, such as expert review dataassociated with a review of modified output data 450 by a panel of humanexperts. This way, at 540, quality evaluation of modified output data450 may be performed based on any combination of quality indicator data(Q1, . . . , QK) discussed above. For example, if the credibility scoreof a planner is lower than a particular threshold, the modification(s)made by the planner may be ignored during continuous learning forefficiency.

At 550, 560 and 570 in FIG. 5, in response to a decision to acceptmodified output data 450, second training data 480 may be generated tofacilitate quality-aware continuous learning. Depending on the desiredimplementation, a case weight (w) may be assigned to modified outputstructure data 450 to influence the continuous learning process. Forexample, the case weight may be assigned based on the credibility scoreC(i) associated with the i^(th) planner 455. A higher C(i) will lead toa higher case weight (w1) to indicate a relatively higher measure ofreliability, whereas a lower C(i) will lead to a lower case weight(w2<w1) to reduce its influence in modified segmentation engine 480.Additionally and/or alternatively, the case weight (w) may be assignedbased on statistical parameter data 461 associated with modified outputstructure data 450. See corresponding 560-570 in FIG. 5.

In another example, a case weight (w) may be a function of severalfactors, including magnitude of change δ (i) made by the i^(th) planner455, credibility score C(i) associated with the i^(th) planner 455, etc.For a planner with a substantially low credibility score, a small changemay be ignored. A big change may be reviewed a panel of human experts.If accepted by the panel, the change may be assigned with a low caseweight (w) before being added to the new training set. For a morecredible planner with a substantially high credibility score, a smallchange made by the planner may be accepted. A big change may be reviewedby the panel, and assigned with a higher case weight (w) if accepted.This way, changes made by a more credible planner will potentially havemore material influence on modified AI engine 490. It should beunderstood that any suitable thresholds may be set to determine whethera planner's credibility score is “low” (e.g., C(i)≤C_(threshold)) or“high” (e.g., C(i)>C_(threshold)), and whether the correspondingmagnitude of change is “small” (e.g., δ(i)≤δ_(threshold)) or “big”(e.g., δ(i)>δ_(threshold)).

Dose Prediction and Other Tasks

Examples of the present disclosure may be implemented for othertreatment planning tasks. FIG. 7 is a schematic diagram illustratingexample quality-aware continuous learning for dose prediction 700. Inthis example, dose prediction engine 720 may be trained using firsttraining data 710 during training phase 701; applied to perform doseprediction during inference phase 702; and updated during continuouslearning phase 703 based on quality evaluation.

During training phase (see 701 in FIG. 7), first training data 710 maybe used to train dose prediction engine 720. First training data 710 mayinclude image and structure data 711 (i.e., training input) and dosedata 712 (i.e., training output) associated with multiple past patients.Dose data 712 (e.g., 3D dose data) may specify dose distributions for atarget (denoted “D_(TAR)”) and an OAR (denoted “D_(OAR)”). For example,in relation to prostate cancer, dose data 712 may specify dosedistributions for a target representing the patient's prostate, and anOAR representing a proximal healthy structure such as rectum or bladder.In practice, dose data 712 may specify the dose distributions for thewhole 3D volume, not just the target and OAR volumes. Dose data 712 mayinclude spatial biological effect data (e.g., fractionation correcteddose) and/or cover only part of the treatment volume. Any additionalinput data may be used to train dose prediction engine 720, such as beamgeometry data associated with the treatment delivery system.

During inference phase (see 702 in FIG. 7), dose prediction engine 720may be used to generate output dose data 740 based on input image andstructure data 730 associated with a particular patient. Dose data 740may specify dose distributions for an OAR (“D_(OAR)”) and a target(“D_(TAR)”). Modification(s) may then be made by treatment planner 755(e.g., dosimetrist) to generate modified output dose data 750 based onany suitable dose prediction practice(s) preferred by the planner. Themodification(s) may be associated with OAR sparing, target coverage,target dose prescription, normal tissue dose, location of dosegradients, steepness of dose gradients, orientation of dose gradients,etc.

During continuous learning phase (see 703 in FIG. 7), quality evaluationmay be performed based on statistical parameter 761 and/or credibilityscore 762 associated with planner 755 responsible for themodification(s). As discussed using FIG. 5, example statistical modelsfor evaluating certain attribute(s) associated with dose data 740/750may be used, such as D20, D50, dose fall-off gradient, etc. Credibilityscore 762 may be generated according to the example in FIG. 6, in whichcase planner-generated dose data D(i, j) may be used.

In response to determination to accept modified output dose data 750 forcontinuous learning based on the quality evaluation (see 770-772 in FIG.7), modified dose prediction engine 790 may be generated by re-trainingdose prediction engine 720 based on second training data 780. Similar tothe example in FIG. 4, second training data 780 may include (input,output) pair in the form of input image and structure data 730 andmodified output dose data 750. Once validated and approved, modifieddose prediction engine 790 may be deployed for use in the next iterationof inference phase 702. If modification is made to output dose datagenerated by modified engine 780, continuous learning phase 703 may berepeated for further improvement.

Besides automatic segmentation in FIG. 4 and dose prediction in FIG. 7,quality-aware continuous learning may be implemented for otherradiotherapy treatment planning tasks, such as treatment delivery dataestimation, treatment outcome prediction, etc. The estimated treatmentdelivery data (i.e., output data) may include structure projection data,fluence map data, etc. For example, an AI engine may be trained toperform structure projection data, such as based on image data,structure data, dose data, or any combination thereof. Structureprojection data may include data relating to beam orientations andmachine trajectories for a treatment delivery system.

In another example, an AI engine may be trained to perform fluence mapestimation, such as 2D fluence maps for a set of beam orientations ortrajectories, machine control point data (e.g., jaw and leaf positions,gantry and couch positions), etc. Fluence maps will be explained furtherusing FIG. 8. Any additional and/or alternative training data may beused, such as field geometry data, monitor units (amount of radiationcounted by machine), quality of plan estimate (acceptable or not), dailydose prescription (output), field size or other machine parameters,couch positions parameters or isocenter position within patient,treatment strategy (use movement control mechanism or not, boost or noboost), treat or no treat decision, etc.

Multi-Technique AI Engines

Examples of the present disclosure may be used to facilitatequality-aware continuous learning for multi-technique AI engines forradiotherapy treatment planning. FIG. 8 is a schematic diagramillustrating example quality-aware continuous learning 800 forradiotherapy treatment planning using a multi-technique AI engine.Similar to the example in FIG. 7, dose prediction engine 820 in FIG. 8is a multi-technique AI engine that is trained to perform doseprediction using multiple techniques. Here, the term “multi-technique AIengine” may refer generally to a single AI engine, or a group ofmultiple AI engines that are trained according to respective techniques.

During training phase (see 801 in FIG. 8), first training data 810 maybe used to train multi-technique dose prediction engine 820 to generatemultiple sets of output data. First training data 810 may include imageand structure data 811 (i.e., training input) and dose data 812 (i.e.,training output) associated with multiple techniques denoted as (T1, T2,T3, T4). Example image data, structure data and dose data explainedusing FIGS. 4-7 are also applicable here and will not be repeated forbrevity.

During inference phase (see 802 in FIG. 8), multi-technique doseprediction engine 820 may be used to generate multiple sets of outputdose data 841-844 based on input image and structure data 830 associatedwith a particular patient. For each technique, dose data 840 may specifydose distributions for any suitable structures (e.g., OAR and target).For example, first set 841 may be generated based on a first technique(e.g., T1=5 fields) for treatment delivery, second set 842 based on asecond technique (e.g., T2=3 fields), third set 843 based on a thirdtechnique (e.g., T3=proton therapy), and fourth set 844 based on afourth technique (e.g., T4=VMAT). In practice, each set of output dosedata may be evaluated based on any suitable factor(s) during inferencephase 802, such as deliverability, adherence to dose prescription,collision, OAR limits, machine parameters, any combination thereof, etc.

In the example in FIG. 8, multiple sets 841-844 generated usingdifferent techniques (T1, T2, T3, T4) are then ranked using a costfunction. In the case of dose prediction, the cost function may be basedon time, complexity, DVH, any combination thereof, etc. In the case ofsegmentation, the cost function may be based on segmentation-relatedparameter(s), such as segmentation mean, etc. In practice, the costfunction may be designed for a problem (representing a mathematicalground truth) in which an optimal solution may be found. This way, sets841-844 may be processed using the cost function and ranked accordingly.Further, a generative adversarial network (GAN), or any other suitablegenerative model, may be set up to create new techniques to exploreoptions for which no previously trained AI engine exists.

The ranked list (see 845) may then be presented to treatment planner 855for selection. Any suitable metric(s) may be presented along with eachtechnique to guide the selection process, such as the cost functionmetrics discussed above. Selection may then be made by treatment planner855 (e.g., dosimetrist) to generate modified output dose data 850. Forexample, if the first technique is selected, first set 841 (i.e., basedon T1=5 fields) may be used as modified output dose data 850. Treatmentplanner 855 may also make any additional modification(s) to first set841. The planner's selection made by used to improve or update theapproach used for ranking multiple sets of output data in the nextiteration (see arrow from 855 to 845 in FIG. 8).

During continuous learning phase (see 803 in FIG. 8), quality evaluationmay be performed based on statistical parameter 861 and/or credibilityscore 862 associated with planner 855. In response to a decision toaccept modified output dose data 850 for continuous learning based onthe quality evaluation (see 870-872 in FIG. 8), modified engine 890 maybe generated by updating or re-training multi-technique engine 820 basedon second training data 880. Similar to the example in FIG. 4, secondtraining data 880 may include (input, output) pair in the form of inputdata 830 and modified output dose data 850 (e.g., first set 841). Oncevalidated and approved, modified dose prediction engine 890 may bedeployed for use in the next iteration of inference phase 802 tofacilitate further improvement. Using the example in FIG. 8,multi-technique comparisons may be performed using AI engines, whichgenerally improves efficiency compared to brute force techniques.

Other Use Cases of Credibility Score

According to examples of the present disclosure, a credibility scoreC(i) may be assigned to a planner to facilitate various aspects ofquality-aware continuous learning. Some additional use cases will bediscussed using FIG. 9, which is a flowchart of example process 900 forrequest processing based on credibility score. Example process 900 mayinclude one or more operations, functions, or actions illustrated by oneor more blocks, such as 910 to 931. The various blocks may be combinedinto fewer blocks, divided into additional blocks, and/or eliminatedbased upon the desired implementation. Example process 900 may beimplemented using any suitable computer system(s), an example of whichwill be discussed using FIG. 11. The computer system may be configuredto perform example process 900 according to a user's request to performany one of the following: expert panel selection, planning taskassignment, reward determination, etc. The user's request may begenerated using any suitable user interface, such as applicationprogramming interface (API), graphical user interface (GUI), commandline interface (CLI), etc.

In relation to expert panel selection (see 910-915 in FIG. 9), thecomputer system may be configured to select, from multiple treatmentplanners, a panel of “experts” to review modified output data (e.g.,350/450/750) based on their credibility score C(i). This way, an expertpanel may be assembled periodically to perform quality evaluation byreviewing modification(s) made by other planners, and deciding whetherto accept the modification(s). Depending on the credibility score, theexpert panel may be updated over time to reflect changing practices.

In one example, in response to receiving a request for expert panelselection, a planner with the relevant expertise may be identified andselected based on their credibility score C(i). For example, a selectedplanner may be one whose quality metric seems to have been violated themost by the modification(s) in the modified output data. Each expert(i.e., selected planner) is then requested to review the modified outputdata, such as by sending the expert an anonymized snapshot of themodification(s). After performing an offline review, each expert maysubmit their individual decision (e.g., vote) as to whether to accept orreject. A final decision may be made based on review decisions submittedby different experts. If accepted, the modified output data will be usedas part of training data for continuous learning purposes.

In relation to task assignment (see 920-922 in FIG. 9), certain planningtasks may be assigned to certain planners based on their credibilityscore. In this case, the computer system may be configured to select,from multiple planners, a particular planner to perform a particularplanning task based on their credibility score C(i). In one example, inresponse to receiving a request for task assignment, planner(s) withrelevant expertise relating to the planning task may be identified andselected. For example, a planning task relating to breast cancer mightbe assigned to a planner who has the highest credibility score andexpertise relating to breast cancer treatment. Another planning taskrelating to prostate cancer might be assigned to a different planner whois most credible in prostate cancer treatment.

In relation to reward determination (see 930-931 in FIG. 9), a reward(e.g., monetary reward, promotion, award, etc.) may be determined for aplanner based on their credibility score. For example, a reward R(i) forthe i^(th) planner may be proportional to the planner's C(i). A morecredible planner should receive a better reward compared to a lesscredible planner. The reward may be used as an incentive for planners toimprove their credibility score over time.

Example Treatment Plan

During radiotherapy treatment planning, treatment plan 156/1000 may begenerated based on structure data and/or dose data generated usingtreatment planning engines discussed above. For example, FIG. 10 is aschematic diagram of example treatment plan 156/1000 generated orimproved based on output data in the examples in FIG. 1 to FIG. 9.Treatment plan 156 may be delivered using any suitable treatmentdelivery system that includes radiation source 1010 to project radiationbeam 1020 onto treatment volume 1060 representing the patient's anatomyat various beam angles 1030.

Although not shown in FIG. 10 for simplicity, radiation source 1010 mayinclude a linear accelerator to accelerate radiation beam 1020 and acollimator (e.g., MLC) to modify or modulate radiation beam 1020. Inanother example, radiation beam 1020 may be modulated by scanning itacross a target patient in a specific pattern with various energies anddwell times (e.g., as in proton therapy). A controller (e.g., computersystem) may be used to control the operation of radiation source 1020according to treatment plan 156.

During treatment delivery, radiation source 1010 may be rotatable usinga gantry around a patient, or the patient may be rotated (as in someproton radiotherapy solutions) to emit radiation beam 1020 at variousbeam orientations or angles relative to the patient. For example, fiveequally-spaced beam angles 1030A-E (also labelled “A,” “B,” “C,” “D” and“E”) may be selected using an AI engine configured to perform treatmentdelivery data estimation. In practice, any suitable number of beamand/or table or chair angles 1030 (e.g., five, seven, etc.) may beselected. At each beam angle, radiation beam 1020 is associated withfluence plane 1040 (also known as an intersection plane) situatedoutside the patient envelope along a beam axis extending from radiationsource 1010 to treatment volume 1060. As shown in FIG. 10, fluence plane1040 is generally at a known distance from the isocenter.

In addition to beam angles 1030A-E, fluence parameters of radiation beam1020 are required for treatment delivery. The term “fluence parameters”may refer generally to characteristics of radiation beam 1020, such asits intensity profile as represented using fluence maps (e.g., 1050A-Efor corresponding beam angles 1030A-E). Each fluence map (e.g., 1050A)represents the intensity of radiation beam 1020 at each point on fluenceplane 1040 at a particular beam angle (e.g., 1030A). Treatment deliverymay then be performed according to fluence maps 1050A-E, such as usingIMRT, etc. The radiation dose deposited according to fluence maps1050A-E should, as much as possible, correspond to the treatment plangenerated according to examples of the present disclosure.

Computer System

The above examples can be implemented by hardware, software or firmwareor a combination thereof. FIG. 11 is a schematic diagram of examplecomputer system 1100 for quality-aware continuous learning forradiotherapy treatment planning. In this example, computer system 1105(also known as a treatment planning system) may include processor 1110,computer-readable storage medium 1120, interface 1140 to interface withradiotherapy treatment delivery system 160, and bus 1130 thatfacilitates communication among these illustrated components and othercomponents.

Processor 1110 is to perform processes described herein with referenceto FIG. 1 to FIG. 9. Computer-readable storage medium 1120 may store anysuitable information 1122, such as information relating to trainingdata, AI engines, weight data, input data, output data, etc.Computer-readable storage medium 1120 may further storecomputer-readable instructions 1124 which, in response to execution byprocessor 1110, cause processor 1110 to perform processes describedherein. Treatment may be delivered according to treatment plan 156 usingtreatment planning system 160 explained using FIG. 1, the description ofwhich will not be repeated here for brevity.

The foregoing detailed description has set forth various embodiments ofthe devices and/or processes via the use of block diagrams, flowcharts,and/or examples. Insofar as such block diagrams, flowcharts, and/orexamples contain one or more functions and/or operations, it will beunderstood by those within the art that each function and/or operationwithin such block diagrams, flowcharts, or examples can be implemented,individually and/or collectively, by a wide range of hardware, software,firmware, or virtually any combination thereof. Throughout the presentdisclosure, the terms “first,” “second,” “third,” etc. do not denote anyorder of importance, but are rather used to distinguish one element fromanother.

Those skilled in the art will recognize that some aspects of theembodiments disclosed herein, in whole or in part, can be equivalentlyimplemented in integrated circuits, as one or more computer programsrunning on one or more computers (e.g., as one or more programs runningon one or more computer systems), as one or more programs running on oneor more processors (e.g., as one or more programs running on one or moremicroprocessors), as firmware, or as virtually any combination thereof,and that designing the circuitry and/or writing the code for thesoftware and or firmware would be well within the skill of one of skillin the art in light of this disclosure.

Although the present disclosure has been described with reference tospecific exemplary embodiments, it will be recognized that thedisclosure is not limited to the embodiments described, but can bepracticed with modification and alteration within the spirit and scopeof the appended claims. Accordingly, the specification and drawings areto be regarded in an illustrative sense rather than a restrictive sense.

We claim:
 1. A method for a computer system to perform quality-awarecontinuous learning for radiotherapy treatment planning, wherein themethod comprises: obtaining an artificial intelligence (AI) engine thatis trained to perform a radiotherapy treatment planning task; based oninput data associated with a patient, performing the radiotherapytreatment planning task using the AI engine to generate output dataassociated with the patient; obtaining modified output data thatincludes one or more modifications made by a treatment planner to theoutput data; performing quality evaluation based on at least one of (a)first quality indicator data associated with the modified output data,and (b) second quality indicator data associated with the treatmentplanner; and in response to a decision to accept the modified outputdata based on the quality evaluation, generating a modified AI engine byre-training the AI engine based on the modified output data.
 2. Themethod of claim 1, wherein performing quality evaluation comprises:determining the first quality indicator data in the form of statisticalparameter data associated with the modified output data by applying oneor more statistical models on the modified output data.
 3. The method ofclaim 1, wherein performing quality evaluation comprises: identifyingthe treatment planner from multiple treatment planners; and determiningthe second quality indicator data in the form of a credibility scoreassigned to the treatment planner.
 4. The method of claim 1, whereingenerating the modified AI engine comprises: assigning a case weight tothe modified output data based on at least one of (a) the first qualityindicator data, and (b) the second quality indicator data.
 5. The methodof claim 1, wherein the method further comprises: obtainingplanner-generated output data that is generated by the multipletreatment planners based on multiple treatment planning cases; based onthe planner-generated output data, determining consensus truth dataassociated with the multiple treatment planning cases; and based on acomparison between the planner-generated output data and the consensustruth data, assigning the second quality indicator data in the form of acredibility score to each of the multiple treatment planners.
 6. Themethod of claim 1, wherein the method further comprises: performing theradiotherapy treatment planning task by using the AI engine to generatemultiple sets of output data based on respective multiple techniquesassociated with the radiotherapy treatment planning task; and obtainingmodified output data in the form of a particular set of output data thatis selected by the treatment planner from the multiple sets, wherein theparticular set of output data is generated based on one of the multipletechniques.
 7. The method of claim 1, wherein the method furthercomprises one of the following: selecting, from multiple treatmentplanners, a panel of experts to review the modified output data based onmultiple credibility scores assigned to the respective multipletreatment planners; selecting, from multiple treatment planners, aparticular treatment planner to a particular radiotherapy treatmentplanning task based on a particular credibility score assigned to theparticular treatment planner; and determining, for a particulartreatment planner, a reward based on a particular credibility scoreassigned to the particular treatment planner.
 8. The method of claim 1,wherein performing the radiotherapy treatment planning task comprises:performing automatic segmentation using the AI engine to generate outputstructure data based on input image data associated with the particularpatient; performing dose prediction using the AI engine to generateoutput dose data based on input image data and input structure dataassociated with the particular patient; and performing treatmentdelivery data prediction using the AI engine to generate treatmentdelivery data based on input dose data associated with the particularpatient.
 9. A non-transitory computer-readable storage medium thatincludes a set of instructions which, in response to execution by aprocessor of a computer system, cause the processor to perform a methodof quality-aware continuous learning for radiotherapy treatmentplanning, wherein the method comprises: obtaining an artificialintelligence (AI) engine that is trained to perform a radiotherapytreatment planning task; based on input data associated with a patient,performing the radiotherapy treatment planning task using the AI engineto generate output data associated with the patient; obtaining modifiedoutput data that includes one or more modifications made by a treatmentplanner to the output data; performing quality evaluation based on atleast one of (a) first quality indicator data associated with themodified output data, and (b) second quality indicator data associatedwith the treatment planner; and in response to a decision to accept themodified output data based on the quality evaluation, generating amodified AI engine by re-training the AI engine based on the modifiedoutput data.
 10. The non-transitory computer-readable storage medium ofclaim 9, wherein performing quality evaluation comprises: determiningthe first quality indicator data in the form of statistical parameterdata associated with the modified output data by applying one or morestatistical models on the modified output data.
 11. The non-transitorycomputer-readable storage medium of claim 9, wherein performing qualityevaluation comprises: identifying the treatment planner from multipletreatment planners; and determining the second quality indicator data inthe form of a credibility score assigned to the treatment planner. 12.The non-transitory computer-readable storage medium of claim 9, whereingenerating the modified AI engine comprises: assigning a case weight tothe modified output data based on at least one of (a) the first qualityindicator data, and (b) the second quality indicator data.
 13. Thenon-transitory computer-readable storage medium of claim 9, wherein themethod further comprises: obtaining planner-generated output data thatis generated by the multiple treatment planners based on multipletreatment planning cases; based on the planner-generated output data,determining consensus truth data associated with the multiple treatmentplanning cases; and based on a comparison between the planner-generatedoutput data and the consensus truth data, assigning the second qualityindicator data in the form of a credibility score to each of themultiple treatment planners.
 14. The non-transitory computer-readablestorage medium of claim 9, wherein the method further comprises:performing the radiotherapy treatment planning task by using the AIengine to generate multiple sets of output data based on respectivemultiple techniques associated with the radiotherapy treatment planningtask; and obtaining modified output data in the form of a particular setof output data that is selected by the treatment planner from themultiple sets, wherein the particular set of output data is generatedbased on one of the multiple techniques.
 15. The non-transitorycomputer-readable storage medium of claim 9, wherein the method furthercomprises one of the following: selecting, from multiple treatmentplanners, a panel of experts to review the modified output data based onmultiple credibility scores assigned to the respective multipletreatment planners; selecting, from multiple treatment planners, aparticular treatment planner to a particular radiotherapy treatmentplanning task based on a particular credibility score assigned to theparticular treatment planner; and determining, for a particulartreatment planner, a reward based on a particular credibility scoreassigned to the particular treatment planner.
 16. The non-transitorycomputer-readable storage medium of claim 9, wherein performing theradiotherapy treatment planning task comprises: performing automaticsegmentation using the AI engine to generate output structure data basedon input image data associated with the particular patient; performingdose prediction using the AI engine to generate output dose data basedon input image data and input structure data associated with theparticular patient; and performing treatment delivery data predictionusing the AI engine to generate treatment delivery data based on inputdose data associated with the particular patient.
 17. A computer systemconfigured to perform quality-aware continuous learning for radiotherapytreatment planning, wherein the computer system comprises: a processorand a non-transitory computer-readable medium having stored thereoninstructions that, when executed by the processor, cause the processorto: obtain an artificial intelligence (AI) engine that is trained toperform a radiotherapy treatment planning task; based on input dataassociated with a patient, perform the radiotherapy treatment planningtask using the AI engine to generate output data associated with thepatient; obtain modified output data that includes one or moremodifications made by a treatment planner to the output data; performquality evaluation based on at least one of (a) first quality indicatordata associated with the modified output data, and (b) second qualityindicator data associated with the treatment planner; and in response toa decision to accept the modified output data based on the qualityevaluation, generate a modified AI engine by re-training the AI enginebased on the modified output data.
 18. The computer system of claim 17,wherein the instructions for performing quality evaluation cause theprocessor to: determine the first quality indicator data in the form ofstatistical parameter data associated with the modified output data byapplying one or more statistical models on the modified output data. 19.The computer system of claim 17, wherein the instructions for performingquality evaluation cause the processor to: identify the treatmentplanner from multiple treatment planners; and determine the secondquality indicator data in the form of a credibility score assigned tothe treatment planner.
 20. The computer system of claim 17, wherein theinstructions for generating the modified AI engine cause the processorto: assign a case weight to the modified output data based on at leastone of (a) the first quality indicator data, and (b) the second qualityindicator data.
 21. The computer system of claim 17, wherein theinstructions further cause the processor to: obtain planner-generatedoutput data that is generated by the multiple treatment planners basedon multiple treatment planning cases; based on the planner-generatedoutput data, determine consensus truth data associated with the multipletreatment planning cases; and based on a comparison between theplanner-generated output data and the consensus truth data, assign thesecond quality indicator data in the form of a credibility score to eachof the multiple treatment planners.
 22. The computer system of claim 17,wherein the instructions further cause the processor to: perform theradiotherapy treatment planning task by using the AI engine to generatemultiple sets of output data based on respective multiple techniquesassociated with the radiotherapy treatment planning task; and obtainmodified output data in the form of a particular set of output data thatis selected by the treatment planner from the multiple sets, wherein theparticular set of output data is generated based on one of the multipletechniques.
 23. The computer system of claim 17, wherein theinstructions further cause the processor to perform one of thefollowing: select, from multiple treatment planners, a panel of expertsto review the modified output data based on multiple credibility scoresassigned to the respective multiple treatment planners; select, frommultiple treatment planners, a particular treatment planner to aparticular radiotherapy treatment planning task based on a particularcredibility score assigned to the particular treatment planner; anddetermine, for a particular treatment planner, a reward based on aparticular credibility score assigned to the particular treatmentplanner.
 24. The computer system of claim 17, wherein the instructionsfor performing the radiotherapy treatment planning task cause theprocessor to: perform automatic segmentation using the AI engine togenerate output structure data based on input image data associated withthe particular patient; perform dose prediction using the AI engine togenerate output dose data based on input image data and input structuredata associated with the particular patient; and perform treatmentdelivery data prediction using the AI engine to generate treatmentdelivery data based on input dose data associated with the particularpatient.